MPAS dycore advancements by dustinswales · Pull Request #3156 · ufs-community/ufs-weather-model

dustinswales · 2026-03-20T16:05:06Z

Commit Queue Requirements:

This PR addresses a relevant WM issue (if not, create an issue).
All subcomponent pull requests (if any) have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines), preferably on Ursa (Derecho or Hercules are acceptable alternatives). Exceptions: documentation-only PRs, CI-only PRs, etc.
- Commit log file w/full results from RT suite run (if applicable).
- Verify that test_changes.list indicates which tests, if any, are changed by this PR. Commit test_changes.list, even if it is empty.
Fill out all sections of this template.

Description:

Updates to the MPAS dynamical core interface to the UFSATM.

New advancements within th UFSATM:

Constituent handling and scalar advection. New module, ufs_mpas_constituents, which contains interface code between MPAS constituents and the UFS.
MPAS native output stream. Contained within the ufs_mpas_io module. This module also includes procedures for reading in the various MPAS input streams (e.g., invariant, input, sfc_input)
Support for lateral boundary condition (LBC) stream. New module with code related to regional MPAS configurations, ufs_mpas_boundaries. The LBCs are updated during forecast integration, _run phase in ufs_mpas_subdriver.F90.
Progress on CCPP physics coupling. New MPAS input stream (sfc_input) for gridded fields needed by the physics. Additional P2D coupling procedures to handle the state evolution between physics group and dycore calls. Added diagnostics to output stream.

Within the UWM, the changes are entirely to the MPAS testing configurations within the UWM.

Commit Message:

* UFSWM - Updates to the MPAS dynamical core interface to the UFSATM.
  * UFSATM - Updates to the MPAS dynamical core interface to the UFSATM.
    * ccpp-physics - Small bug fix for photochemistry.

Priority:

Critical Bugfix: Reason
High: Reason
Normal

Git Tracking

UFSWM:

Closes MPAS Lateral Boundary Condition Stream #2782
Closes Add native MPAS output #2879
Working towards CCPP Physics to MPAS dynamics coupling #2781

Sub component Pull Requests:

UFSATM: Some more MPAS functionality NOAA-EMC/ufsatm#1079
- ccpp-physics: Small bug when not using photochemistry ccpp-physics#365

UFSWM Blocking Dependencies:

Blocked by #
None

Documentation:

Documentation update required.
- Relevant updates are included with this PR.
- A WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
Documentation update NOT required.
- Explanation:

Changes

Regression Test Changes (Please commit test_changes.list):

PR Adds New Tests/Baselines.
PR Updates/Changes Baselines.
No Baseline Changes.

Input data Changes:

None.
PR adds input data.
PR changes existing input data.

Library Changes/Upgrades:

Required
- Library names w/versions:
- Git Stack Issue (JCSDA/spack-stack#)
No Updates

Testing Log:

…grate over correct forecast interval.

…odel into fix/derecho

…r-model into feature/mpas_lbc_plus_cleanup

…stinswales/ufs-weather-model into feature/mpas_lbc_plus_cleanup

gspetro-NOAA · 2026-04-27T18:57:38Z

@dustinswales Baseline creation ran successfully on Hercules, but on Derecho, I am seeing errors that are causing the baseline creation to fail. Errors are similar for both tests, so here's what I see for control_gfs_mpas_gnu:

The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
Abort with message Unknown error in file operation in file /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/cache/build_stage/spack-stage-parallelio-2.6.2-m4egtstkuepe7z7mkthzab2mdp3aataj/spack-src/src/clib/pioc_support.c at line 2176
Abort with message Unknown error in file operation in file /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/cache/build_stage/spack-stage-parallelio-2.6.2-m4egtstkuepe7z7mkthzab2mdp3aataj/spack-src/src/clib/pioc_support.c at line 2176
Obtained 10 stack frames.
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(print_trace+0x24) [0x15168620fc54]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(piodie+0x3f) [0x15168620fd0f]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(check_netcdf2+0x149) [0x151686210049]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(PIOc_createfile_int+0x4b1) [0x151686214491]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(PIOc_createfile+0x3f) [0x15168620f11f]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpiof.so(__piolib_mod_MOD_createfile+0xaa) [0x151685c123aa]
./fv3.exe() [0x12990d8]
./fv3.exe() [0x12824f9]
./fv3.exe() [0x127ae80]
./fv3.exe() [0x12578e9]
MPICH ERROR [Rank 1] [job id 0ae03ea6-9e15-4cee-b2be-623f984cf68b] [Mon Apr 27 12:45:21 2026] [dec1239] - Abort(-1) (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1
Abort with message Unknown error in file operation in file /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/cache/build_stage/spack-stage-parallelio-2.6.2-m4egtstkuepe7z7mkthzab2mdp3aataj/spack-src/src/clib/pioc_support.c at line 2176
...

My run_dir is: /glade/derecho/scratch/gpetro/FV3_RT/rt_110427, and my test_dir is: /glade/derecho/scratch/gpetro/RTs/ufs-wm/3156a/tests.

on-behalf-of @ufs-community <[email protected]>

dustinswales · 2026-04-27T19:53:39Z

@dustinswales Baseline creation ran successfully on Hercules, but on Derecho, I am seeing errors that are causing the baseline creation to fail. Errors are similar for both tests, so here's what I see for control_gfs_mpas_gnu:

The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
The specified filesystem prefix is not supported in this library:
    history.2023-03-10_15:00:00.nc
Abort with message Unknown error in file operation in file /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/cache/build_stage/spack-stage-parallelio-2.6.2-m4egtstkuepe7z7mkthzab2mdp3aataj/spack-src/src/clib/pioc_support.c at line 2176
Abort with message Unknown error in file operation in file /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/cache/build_stage/spack-stage-parallelio-2.6.2-m4egtstkuepe7z7mkthzab2mdp3aataj/spack-src/src/clib/pioc_support.c at line 2176
Obtained 10 stack frames.
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(print_trace+0x24) [0x15168620fc54]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(piodie+0x3f) [0x15168620fd0f]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(check_netcdf2+0x149) [0x151686210049]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(PIOc_createfile_int+0x4b1) [0x151686214491]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpioc.so(PIOc_createfile+0x3f) [0x15168620f11f]
/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/envs/ue-gcc-12.4/install/gcc/12.4.0/parallelio-2.6.2-m4egtst/lib/libpiof.so(__piolib_mod_MOD_createfile+0xaa) [0x151685c123aa]
./fv3.exe() [0x12990d8]
./fv3.exe() [0x12824f9]
./fv3.exe() [0x127ae80]
./fv3.exe() [0x12578e9]
MPICH ERROR [Rank 1] [job id 0ae03ea6-9e15-4cee-b2be-623f984cf68b] [Mon Apr 27 12:45:21 2026] [dec1239] - Abort(-1) (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1
Abort with message Unknown error in file operation in file /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.9.2/cache/build_stage/spack-stage-parallelio-2.6.2-m4egtstkuepe7z7mkthzab2mdp3aataj/spack-src/src/clib/pioc_support.c at line 2176
...

My run_dir is: /glade/derecho/scratch/gpetro/FV3_RT/rt_110427, and my test_dir is: /glade/derecho/scratch/gpetro/RTs/ufs-wm/3156a/tests.

Hmmm. That's odd.
@scrasmussen Do you have any ideas/insight on this?

gspetro-NOAA · 2026-04-28T15:57:54Z

@dustinswales Any update on the Derecho troubleshooting? (Not sure if anything is happening offline.) If you don't think this problem can be solved today, we can remove derecho from the machine list, file an issue, and you can put in a fix with the next PR (or issue a bugfix PR).

dustinswales · 2026-04-28T16:33:38Z

@dustinswales Any update on the Derecho troubleshooting? (Not sure if anything is happening offline.) If you don't think this problem can be solved today, we can remove derecho from the machine list, file an issue, and you can put in a fix with the next PR (or issue a bugfix PR).

@gspetro-NOAA I just figured out the issue on Derecho. The default PIO output type is PNETCDF, which doesn't work on Derecho. If I change the default PIO output type to NETCDF it runs on Derecho.

I agree with your suggestion. (Remove Derecho, open issue, and I'll add to #3201)

For references, here are the changes causing trouble in the UFSATM NUOPC Cap:

-       cvalue = 'NETCDF'
-       pio_iotype = PIO_IOTYPE_NETCDF
+       cvalue = 'PNETCDF'
+       pio_iotype = PIO_IOTYPE_PNETCDF

gspetro-NOAA · 2026-04-29T13:15:26Z

Testing has concluded successfully; leaving a note in sub-PRs.

…stinswales/ufs-weather-model into feature/mpas_lbc_plus_cleanup

gspetro-NOAA · 2026-04-29T18:45:50Z

@dustinswales Please revert .gitmodules and update the UFSATM hash to 967cba8.

Natalie Perlin and others added 30 commits August 12, 2025 13:53

Derecho update of rocoto and ecflow paths

533f940

Derecho rt_utils.sh: wait longer to query completed job status

232f96b

update derecho job_card fv3_qsub.IN_derecho

8056a36

Derecho: updated spack-stack, job_card, RT tests configurations

c1a3007

Merge branch 'develop' into fix/derecho

8076a14

use new Derecho spack-stack 1.9.2 environment built with ncarenv/24.12

92149f4

update job_card for Derecho

c16d2bb

update for ecflow path on Derecho

d7fdfbc

Merge remote-tracking branch 'origin2/fix/derecho' into fix/derecho

d23b617

Merge remote-tracking branch 'origin/develop' into fix/derecho

290bd0d

New Derecho environment with spack-stack-1.9.2

913f2e6

Merge branch 'ufs-community:develop' into fix/derecho

9b26d36

updates of modulefile and compile_qsub for derecho

5af4b25

Merge recent changes from develop branch into fix/derecho

93e19fb

update a script fv3_qsub.IN_derecho

0c638f2

updates for derecho RTs

6faed91

Merge branch 'develop' into fix/derecho

9016e69

Some cleanup. Started adding LBC stream. Updates to NUOPC cap to inte…

80c0ca7

…grate over correct forecast interval.

Merge branch 'develop' into fix/derecho

d995154

Merge branch 'develop' into fix/derecho

e958136

tests configurations updated for derecho

793f072

Merge branch 'fix/derecho' of github.com:natalie-perlin/ufs-weather-m…

2be10e2

…odel into fix/derecho

update UFSATM

00e03af

Merge branch 'ufs-community:develop' into fix/derecho

c3ce609

updated module-setup.sh for Derecho and ufs_derecho.intel.lua modulefile

25fc5a9

revert changes in fv3_qsub.IN_derecho to form a job_card

2ed1ff8

Sync ATM

527a666

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

dac2b1b

…r-model into feature/mpas_lbc_plus_cleanup

Merge branch 'feature/mpas_lbc_plus_cleanup' of https://github.com/du…

20575b7

…stinswales/ufs-weather-model into feature/mpas_lbc_plus_cleanup

Changes to decomposition in P2D. MPAS RTs on URSA now working.

433e079

FernandoAndrade-NOAA added the ursa-ORT label Apr 27, 2026

ORT Jobs Completed.

ad4c41a

on-behalf-of @ufs-community <[email protected]>

epic-cicd-jenkins removed the ursa-ORT label Apr 27, 2026

gspetro-NOAA and others added 2 commits April 27, 2026 15:18

hercules rt log - passed

c700909

add ursa RT log passed

f226ad0

dustinswales mentioned this pull request Apr 28, 2026

MPAS RTs on NCARs Derecho #3209

Open

gspetro-NOAA added 3 commits April 28, 2026 12:27

remove derecho from mpas tests

51131ec

rm derecho from mpas debug test

f181832

derecho rt log - passed

87866a7

gspetro-NOAA removed the In Testing The PR that is currently in testing stages label Apr 29, 2026

dustinswales added 2 commits April 29, 2026 17:06

Update physics hash and .gitmodule

f0f727e

Merge branch 'feature/mpas_lbc_plus_cleanup' of https://github.com/du…

c4c904c

…stinswales/ufs-weather-model into feature/mpas_lbc_plus_cleanup

Update ATM hash. Revert .gitmodules

91d09a6

gspetro-NOAA reviewed Apr 29, 2026

View reviewed changes

Comment thread .gitmodules Outdated

doh!

a2276db

gspetro-NOAA approved these changes Apr 29, 2026

View reviewed changes

gspetro-NOAA requested review from BrianCurtis-NOAA, FernandoAndrade-NOAA, NickSzapiro-NOAA, dpsarmie and jkbk2004 April 29, 2026 19:26

jkbk2004 approved these changes Apr 29, 2026

View reviewed changes

BrianCurtis-NOAA approved these changes Apr 29, 2026

View reviewed changes

gspetro-NOAA merged commit 31863b9 into ufs-community:develop Apr 29, 2026
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MPAS dycore advancements#3156

MPAS dycore advancements#3156
gspetro-NOAA merged 104 commits intoufs-community:developfrom
dustinswales:feature/mpas_lbc_plus_cleanup

dustinswales commented Mar 20, 2026 •

edited by dpsarmie

Loading

Uh oh!

gspetro-NOAA commented Apr 27, 2026

Uh oh!

dustinswales commented Apr 27, 2026

Uh oh!

gspetro-NOAA commented Apr 28, 2026

Uh oh!

dustinswales commented Apr 28, 2026 •

edited

Loading

Uh oh!

gspetro-NOAA commented Apr 29, 2026

Uh oh!

gspetro-NOAA commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

dustinswales commented Mar 20, 2026 • edited by dpsarmie Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Documentation:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

Uh oh!

gspetro-NOAA commented Apr 27, 2026

Uh oh!

dustinswales commented Apr 27, 2026

Uh oh!

gspetro-NOAA commented Apr 28, 2026

Uh oh!

dustinswales commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gspetro-NOAA commented Apr 29, 2026

Uh oh!

gspetro-NOAA commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

dustinswales commented Mar 20, 2026 •

edited by dpsarmie

Loading

dustinswales commented Apr 28, 2026 •

edited

Loading